Skip to content

Fix SuppressStdoutStderr permanently silencing the process when contexts overlap - #3290

Merged
dennisbader merged 4 commits into
unit8co:masterfrom
choksi2212:fix/suppress-stdout-thread-safety-3276
Oct 28, 2026
Merged

dennisbader merged 4 commits into
unit8co:masterfrom
choksi2212:fix/suppress-stdout-thread-safety-3276

Conversation

@choksi2212

Copy link
Copy Markdown
Contributor

SuppressStdoutStderr used to duplicate fd 1 and 2 in __init__ and restore them in __exit__, per instance, with no synchronization. If a second context was created while a first one was active (two threads fitting Prophet models side by side, or just overlapping with blocks), the second one saved the devnull descriptors as if they were the real ones, and whoever exited last pointed fd 1 and 2 back at devnull for the rest of the process lifetime — the scenario from #3276, where after two overlapping fits everything printed afterwards was gone. Creating the context in interpreters without usable standard streams (pythonw on Windows, GUI-embedded hosts) also raised OSError: Bad file descriptor straight from __init__, leaking the descriptors opened just before the failure.

The redirection is now shared state: a module-level lock plus a depth counter, where the outermost context captures and restores the real descriptors, contexts entered while a redirection is already active just join it, and all descriptor work happens in __enter__ with a graceful no-op fallback when the standard streams cannot be saved. I left Prophet's suppression path as-is — it becomes safe through the context manager itself, and switching it over to cmdstanpy logger levels felt like a separate design call.

How I checked it: on master, the new threaded test fails exactly like the issue describes (assert b'' == b'survivor' — the write after both contexts lands nowhere) and the fallback test dies with OSError: [Errno 9] Bad file descriptor out of __init__. With the fix both pass, along with the round-trip and nested tests and the rest of test_logging.py (8 passed). The overlap choreography is event-driven rather than sleep-based, so the interleaving is deterministic. ruff check and ruff format (0.7.2, same as pre-commit) are clean.

Closes #3276.

The context manager duplicated and restored the process-level stdout and
stderr descriptors per instance, with no synchronization. A second context
created while a first one was active saved the devnull descriptors as if
they were the real ones, and whoever exited last dup2-ed devnull back onto
fd 1 and 2, permanently silencing the process (unit8co#3276). Constructing it in
interpreters without real standard streams (pythonw, GUI hosts) also raised
OSError from __init__ and leaked the descriptors opened before the failure.

The redirection is now shared: a module-level lock plus a depth counter, the
outermost context captures and restores the real descriptors, contexts
entered while a redirection is active just join it, and all fd work moved
into __enter__ with a graceful no-op fallback when the standard streams
cannot be saved.

Signed-off-by: choksi2212 <choksi2212@users.noreply.github.com>
Signed-off-by: choksi2212 <choksi2212@users.noreply.github.com>
@codecov

codecov Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.15%. Comparing base (0880bff) to head (2927ee5).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3290   +/-   ##
=======================================
  Coverage   97.14%   97.15%           
=======================================
  Files         175      175           
  Lines       19429    19468   +39     
=======================================
+ Hits        18875    18914   +39     
  Misses        554      554           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions

Copy link
Copy Markdown

Contribution gate — action required

This check failed because None of the linked issues have the ok label applied by a maintainer.

Before this PR can merge:

  1. Open or use an existing GitHub issue and discuss the approach with maintainers.
  2. Wait for a maintainer to add the ok label on that issue.
  3. Link the issue in this PR description (not only the sidebar), e.g. Fixes #<issue> when the PR completes the issue, or Addresses #<issue> / Refs #<issue> for partial work.

See CONTRIBUTING.md and AGENTS.md.

@dennisbader dennisbader left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the PR @choksi2212. LGTM 🚀

@dennisbader
dennisbader merged commit 580358e into unit8co:master Oct 28, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] SuppressStdoutStderr in Prophet.fit() is not thread-safe and permanently redirects process FD 1/2 to /dev/null under concurrent execution

2 participants